Network trouble with Hetzner subnet

[barnybla]

Hi all,
I have an Hetzner ax41 installed with proxmox and ipfire as the router vm. I have additional to the main ip a single ip for the ipfire and a /29 subnet. I can reach the ipfire, but no vm from the subnet. I got a mail from Hetzner abuse, about not allowed MAC addresses.
So my network config is:

auto lo
iface lo inet loopback

auto enp8s0
iface enp8s0 inet manual
        mtu 1500

auto vmbr0
iface vmbr0 inet static
        address xxx.yy.131.13/26 # main ip
        gateway xxx.yy.131.1
        bridge-ports enp8s0
        bridge-stp off
        bridge-fd 0
        mtu 1500
        bridge_maxwait 0
        up ip addr add  ccc.dd.ggg.24/29 dev vmbr0    # subnet

auto vmbr2
iface vmbr2 inet static
        address 192.168.1.1/24
        ovs_type OVSBridge
        ovs_mtu 1500
#green

auto vmbr3
iface vmbr3 inet static
        address 192.168.240.1/24
        ovs_type OVSBridge
        ovs_mtu 1500
#orange

source /etc/network/interfaces.d/*

ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether a8:a1:59:37:de:c2 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:a1:59:37:de:c2 brd ff:ff:ff:ff:ff:ff
    inet zzz.vv.rrr.13/26 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet ccc.dd.ggg.24/29 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet zzz.vv.rrr.12/26 scope global secondary vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aaa1:59ff:fe37:dec2/64 scope link
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:c6:17:2e:2e:86 brd ff:ff:ff:ff:ff:ff
5: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether be:c1:b6:bd:b6:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 scope global vmbr2
       valid_lft forever preferred_lft forever
    inet6 fe80::bcc1:b6ff:febd:b647/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether a2:20:9a:d3:f5:49 brd ff:ff:ff:ff:ff:ff
    inet 192.168.240.1/24 scope global vmbr3
       valid_lft forever preferred_lft forever
    inet6 fe80::a020:9aff:fed3:f549/64 scope link
       valid_lft forever preferred_lft forever
7: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 5a:a2:a3:26:bb:f8 brd ff:ff:ff:ff:ff:ff
8: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether e6:67:7a:5b:89:66 brd ff:ff:ff:ff:ff:ff
9: tap100i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether 66:21:6f:fc:e1:e3 brd ff:ff:ff:ff:ff:ff
10: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether 06:db:5c:f0:2a:7a brd ff:ff:ff:ff:ff:ff
11: tap101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether d2:47:84:a6:b3:85 brd ff:ff:

What is wrong with this config? Why there comming not allowed MAC addresses to the outside?

regards
Bernd

 

[Undergrid]

With regard to the MAC issue...

For individual IP addresses, Hetzner requires you to use specific MAC addresses specified in the "IPs" section of the server information on the Hetzner Robot website. They may have extended this policy for routed subnets (they didn't the last time I had a routed subnet, but it's been a while).

Head over to the Robot site, grab the MAC address for each IP address (if it exists), then set them in the network configuration for you're VM or CT.

The rest of you're configuration looks fine (though it's been a while since I've used a routed subnet on Hetzner) so it might just be the MAC issue, otherwise check firewall settings, IP forwarding etc.

 

[barnybla]

Thanks Undergrid, you are right, it was my fault. I have vergotten to switch on in the Hetzner firewall the subnet. Now it's running!!

sorry
Bernd

 

[silverstone]

I also just received an Abuse message.

But when running `tcpdump -en | grep -i <mac-address-they-say-its-not-allowed>` I get the IP Address of ANOTHER Hetzner Server, NOT Mine !

Did they send the Abuse message to everybody in the /26 Subnet (I have a single IP Address) ?

 

[Undergrid]

I also just received an Abuse message.

But when running `tcpdump -en | grep -i <mac-address-they-say-its-not-allowed>` I get the IP Address of ANOTHER Hetzner Server, NOT Mine !

Did they send the Abuse message to everybody in the /26 Subnet (I have a single IP Address) ?

Are you running PVE 7? There was an issue with 6 and earlier with PVE reflecting messages sent to it with another machines MAC address (IIRC, it's been a while) which causes Hetzner to send abuse messages, but that should have been fixed in 7 (and I haven't seen it since then).

 

[silverstone]

Are you running PVE 7? There was an issue with 6 and earlier with PVE reflecting messages sent to it with another machines MAC address (IIRC, it's been a while) which causes Hetzner to send abuse messages, but that should have been fixed in 7 (and I haven't seen it since then).

I'm on the latest and greatest Proxmox VE 8.2.2.

The only thing I did to try to improve the situation AFTER this Abuse Alert was to add this to /etc/sysctl.d/99-hetzner.conf:

# Ignore ICMP echo packets sent to a broadcast address.
net.ipv4.icmp_echo_ignore_broadcasts=1

# This setting avoids filling up log files with unnecessary error messages coming from invalid responses to broadcast frames.
net.ipv4.icmp_ignore_bogus_error_responses=1

# Disables the acceptance of ICMP redirect messages.
# Make sure no one can modify the routing Table
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0

# Accepting 'secure' ICMP redirects (from those gateways listed as default gateways) has few legitimate uses.
# It should be disabled unless it is absolutely required.
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.all.secure_redirects=0

# A node should not send out IPv4 ICMP redirects, unless it acts as a router.
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0

# Do NOT act as a Router
# IP forwarding should only be enabled on systems acting as routers.
net.ipv4.ip_forward=0
net.ipv6.conf.all.forwarding=0
net.ipv6.conf.default.forwarding=0

# No source routed packets here
# This setting disables the acceptance of packets with the SSR option set in the IPv4 packet header.
# Packets that use Source Routing will be rejected.
# This prevents IP packet redirection, that is redirection to a host behind a firewall, that is not directly reachable otherwise.
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.log_martians=1

# Turn on syncookies for SYN flood attack protection
# This enables TCP SYN Cookie Protection for IPv4 and IPv6.
net.ipv4.tcp_syncookies=1

# Turn on reverse path filter in Strict Mode
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1


# The TCP SYN backlog defines the number of SYN packets that are queued for further processing.
# Once the queue limit is exceeded, all new incoming SYN-packets are dropped and new TCP connections will not be possible (or the SYN cookie protection kicks in)
# Increasing this value improves the protection against TCP SYN flood attacks.
net.ipv4.tcp_max_syn_backlog=4096

But, as I said, I believe the issue is mainly from a "Neighbor" Customer on 2 Servers within the same WAN Subnet.

One of them was associated with a `criminal-ip` hostname for a supposedly cybersecurity company, based on the domain name. I could see that when running `tcpdump -e | grep -i <mac-address-they-say-its-not-allowed>` ...


So I'm not sure. I replied based on my Investigation, updated /etc/sysctl.conf Configuration (based on some guidelines I found online, most reputable I'd say https://documentation.suse.com/es-es/sles/15-SP2/html/SLES-all/cha-sec-sysctl.html, which more or less matched what I found on Reddit ).

The most likely scenario was that my server was contributing/involved to some kind of ICMP flooding (since the `net.ipv4.icmp_echo_ignore_broadcasts` was DISABLED by default I believe, possibly also the ones with `secure_redirects` and `accept_redirects`).
Otherwise it's just me being unlucky of being on the same subnet as someone who was doing something wrong.

As soon as I clicked to "check again" approx. 1h later, I got a new mail saying that everything is fine and "please provide an explanation".


I don't have Firewall enabled on Proxmox VE though, so I wasn't thinking there could be issues.
I thought the Hetzner Firewall (max 10 Rules Inbound + max 10 Rules Outbound) was configured correctly ... I was bitten in the back at least 3 times (both for the Proxmox VE Host and the OPNSense VM), since:
a) Once I changed "Allow All [ICMP]" to "Allow All [Protocols]" ... oups ... got notified by Hetzner that I had rpcbind port 111/udp active ...
b) Once I put a IP Blacklist in place on OPNSense then had a GEO-IP Based "all" bypassing everything else ... oups
c) Port 80 -> Custom Port Redirect in OPNSense was enabled ... oups

I find OPNSense does NOT come with Good Defaults "out of the box". No Access Log enabled, very short amount of versions/backups of Configuration History enabled, HTTPS Strict Disabled, minimum amount of Logs saved [maybe only a few hours ???] etc. So ... difficult to know if anything got compromized or if somebody got access to something. Probably not since everything was updated and Passwords were quite complex.


I'm not saying I was doing everything right, absolutely not ... But sending an email to me when clearly the MAC Address is associated with somebody else ... I mean ... just look at MAC<->IP using tcpdump ...

I didn't enable (yet ?) the Proxmox VE Firewall because well ... I don't want to get locked out. Also on OPNSense I took great care to put in place some anti-lockout rule (basically whitelist my remote IP in OPNSense, also whitelisting all OpenVPN Management Traffic, etc). On Proxmox VE I didn't find it very intuitive to be honest ... https://pve.proxmox.com/wiki/Firewall explains a lot about Configuration Files, but I thought that most of the stuff could havee been done from the GUI.

Furthermore sigh , after iptables deprecation and nftables being forced on everybody, most examples online, GUI tools etc, don't work.

Maybe Saltstack+Firewalld could be an option as an intermediate Layer or nftables-iptables Converter.

EDIT 1: for some reason (maybe because I unlock the system during early boot using `cryptsetup LUKS + dropbear`) I have the `eth0` Interface (Physical Interface of the Server) UP with an IP (and MAC) assigned, while I also have `vmbr0` Bridge Interface UP with the same IP and the same MAC assigned.

 

[silverstone]

So ... I got yet another MAC Abuse warning, this time coupled by a human response, after I inquired about what was going on, since the IP Address related to those MAC Addresses were NOT mine.

They told me that the Server could be locked down in a few hours and to contact Customer Support if that happens.

Before that would happen, I tried to implement the following measures:
- Enable PVE Firewall
- Made /etc/sysctl.d/99-hetzner.conf even more restrictive:

# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# WE MUST ALSO SPECIFY EXPLICITELY eth0 HERE (maybe also vmbr0)
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

# Disable multicast igmp report on local link plug
net.ipv4.igmp_link_local_mcast_reports=0

# Ignore ICMP echo packets sent to a broadcast address. 
net.ipv4.icmp_echo_ignore_broadcasts=1 

# This setting avoids filling up log files with unnecessary error messages coming from invalid responses to broadcast frames. 
net.ipv4.icmp_ignore_bogus_error_responses=1 

# Disable ICMP redirect acceptance and sending to prevent man-in-the-middle attacks
# This will also make PING the Proxmox VE Host Impossible !
net.ipv4.icmp_echo_ignore_all=1

# Disables the acceptance of ICMP redirect messages.
# Make sure no one can modify the routing Table
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
net.ipv4.conf.vmbr0.accept_redirects=0

net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.eth0.accept_redirects=0
net.ipv6.conf.vmbr0.accept_redirects=0

# Accepting 'secure' ICMP redirects (from those gateways listed as default gateways) has few legitimate uses.
# It should be disabled unless it is absolutely required. 
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.eth0.secure_redirects=0
net.ipv4.conf.vmbr0.secure_redirects=0

# A node should not send out IPv4 ICMP redirects, unless it acts as a router. 
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.vmbr0.send_redirects=0

# Do NOT act as a Router
# IP forwarding should only be enabled on systems acting as routers.
net.ipv4.ip_forward=0

net.ipv4.conf.default.forwarding=0
net.ipv4.conf.all.forwarding=0
net.ipv4.conf.eth0.forwarding=0
net.ipv4.conf.vmbr0.forwarding=0

net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.forwarding=0
net.ipv6.conf.eth0.forwarding=0
net.ipv6.conf.vmbr0.forwarding=0

# No source routed packets here
# This setting disables the acceptance of packets with the SSR option set in the IPv4 packet header.
# Packets that use Source Routing will be rejected. 
# This prevents IP packet redirection, that is redirection to a host behind a firewall, that is not directly reachable otherwise. 
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.eth0.accept_source_route=0
net.ipv4.conf.vmbr0.accept_source_route=0

net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.eth0.accept_source_route=0
net.ipv6.conf.vmbr0.accept_source_route=0

# Disable IPv6 Router Advertisements, since they can result in a man-in-the-middle attack
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.eth0.accept_ra=0
net.ipv6.conf.vmbr0.accept_ra=0

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.default.log_martians=1 
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.eth0.log_martians=1 
net.ipv4.conf.vmbr0.log_martians=1 

# Turn on syncookies for SYN flood attack protection
# This enables TCP SYN Cookie Protection for IPv4 and IPv6. 
net.ipv4.tcp_syncookies=1 

# Turn on reverse path filter in Strict Mode
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.eth0.rp_filter=1
net.ipv4.conf.vmbr0.rp_filter=1

# Disable forwarding of all multicast packets on all interfaces
net.ipv4.conf.default.mc_forwarding=0
net.ipv4.conf.all.mc_forwarding=0
net.ipv4.conf.eth0.mc_forwarding=0
net.ipv4.conf.vmbr0.mc_forwarding=0

net.ipv6.conf.default.mc_forwarding=0
net.ipv6.conf.all.mc_forwarding=0
net.ipv6.conf.eth0.mc_forwarding=0
net.ipv6.conf.vmbr0.mc_forwarding=0

# Drop RST packets for sockets in the time-wait state
net.ipv4.tcp_rfc1337=1

# Disable TCP SACK
net.ipv4.tcp_sack=0
net.ipv4.tcp_dsack=0
net.ipv4.tcp_fack=0

# TCP timestamps also leak the system time and should therefore be disabled
net.ipv4.tcp_timestamps=0

# Enable IPv6 Privacy Extensions
net.ipv6.conf.default.use_tempaddr=2
net.ipv6.conf.all.use_tempaddr=2
net.ipv6.conf.eth0.use_tempaddr=2
net.ipv6.conf.vmbr0.use_tempaddr=2

# The TCP SYN backlog defines the number of SYN packets that are queued for further processing. 
# Once the queue limit is exceeded, all new incoming SYN-packets are dropped and new TCP connections will not be possible (or the SYN cookie protection kicks in)
# Increasing this value improves the protection against TCP SYN flood attacks. 
net.ipv4.tcp_max_syn_backlog=4096

Note that you MUST also do net.ipv4.eth0.XXX and net.ipv6.eth0.XXX (possibly also net.ipv4.vmbr0.XXX and net.ipv6.vmbr0.XXX) because I checked (after a reboot) and ... surprise ... eth0 was still NOT following what .all and .default were setting.

Note that by setting net.ipv4.icmp_echo_ignore_all=1 you will GLOBALLY disable ping.

- Updated /etc/network/interfaces

auto lo
iface lo inet loopback

#auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
    hwaddress XXXXXXXXXXXXXX
    address xx.yyy.yyy.zzz
    netmask 255.255.255.255
    gateway xx.yyy.yyy.zzz
    pointopoint xx.yyy.yyy.zzz
    bridge-ports eth0
    bridge-stp off
    bridge_waitport 0
    bridge-fd 0
    bridge-disable-mac-learning 1
    bridge-unicast-flood off
    bridge-multicast-flood off
    bridge-vlan-aware yes
    bridge-vids 2-4096
    pre-up ip addr flush dev eth0
    post-up ip addr flush dev eth0

- Wrote a custom Script /usr/local/bin/disable-network-flooding

#!/bin/bash

# Optional Argument for Verbose Output
verbose=${1-""}

# Disable Unicast flooding
mapfile -t list < <( find /sys -iname *unicast_flood* )

for item in "${list[@]}"
do
    # Echo
    if [[ -n "${verbose}" ]]
    then
       echo "Executing: echo 0 > ${item}"
    fi

    # Disable unicast flooding on all interfaces
    echo 0 > $item
done

# Disable Multicast flooding
#mapfile -t list < <( find /sys -iname *multicast_flood* )
#
#for item in "${list[@]}"
#do
#    # Echo
#    if [[ -n "${verbose}" ]]
#    then
#       echo "Executing: echo 0 > ${item}"
#    fi
#
#    # Disable unicast flooding on all interfaces
#    echo 0 > $item
#done

# Disable Broadcast flooding
#mapfile -t list < <( find /sys -iname *broadcast_flood* )
#
#for item in "${list[@]}"
#do
#    # Echo
#    if [[ -n "${verbose}" ]]
#    then
#       echo "Executing: echo 0 > ${item}"
#    fi
#
#    # Disable unicast flooding on all interfaces
#    echo 0 > $item
#done

(Note that I disabled Multicast/Broadcast because that was causing other issues - to be investigated)

- Created /etc/cron.d/disable-network-flooding

MAILTO=""
*/5 * * * * root /usr/local/bin/disable-network-flooding

- Also called this Script in /etc/rc.local (need a Systemd Service to be able to run rc.local)
- PVE Firewall

In Datacenter -> Firewall -> IP Set -> Create some basic IP Sets / Aliases
hetzner_gateway_ip
management
opnsense_public_ip
proxmox_public_ip

In Datacenter -> Firewall -> Security Group -> Create some basic group of Rules


allow_icmp (only for Proxmox VE Guests, NOT HOST, if net.ipv4.icmp_echo_ignore_all=1):


block_broadcast:


proxmox_access:

destination ports must match your SSH / Web GUI Port.

proxmox_block:




Select your Proxmox VE Node -> Firewall


Select your Proxmox VE Node -> Firewall -> Options


In order to get DHCP Working in the VM some extra steps are needed:
Select VM -> Firewall



Select VM -> Firewall -> Options


Note: the rule for port 67/68 to allow DHCP IP Assignment is REQUIRED. Selecting only DHCP: YES in the Options was NOT enough for my VM to get an IP Address !

I think we might need to make a sticky thread, given how many people are facing this MAC Address Abuse Issue with Hetzner .
I tried to regroup many things I read on this Forum, even though some of them (e.g. port 43) should already have been fixed since Proxmox VE 7.

Nevertheless I still got the Abuse Email .

Let's leave tcpdump run overnight:

tmux
tcpdump -Q out -n -w /root/tcpdump.pcap -e not ether host ab:cd:de:ab:cd:de and not ether host 01:23:45:67:89:01 &
tail -f /root/tcpdump.pcap

 

[Undergrid]

I don't know if this helps, but I have a rule at the datacenter, host and each VM/CT that drops incoming traffic addresses to ip addresses that are not one of the ones assigned to me. (Define and ipset of your assigned addresses then drop anything not addressed to something in that ipset).

I certainly haven't had to do all that stuff you have, though I do use bridged networking rather than routed.

 

[silverstone]

I don't know if this helps, but I have a rule at the datacenter, host and each VM/CT that drops incoming traffic addresses to ip addresses that are not one of the ones assigned to me. (Define and ipset of your assigned addresses then drop anything not addressed to something in that ipset).

I certainly haven't had to do all that stuff you have, though I do use bridged networking rather than routed.

I have bridged networking as well ...

Good point, I'll also add that rule .

EDIT 1: and Hetzner again complained that I did not reply tho their automatic request (because when I clicked they stated that the ticket was already closed). Doesn't make sense ...

EDIT 2: how did you "Invert Match" ? And is Proxmox Firewall First-Match or Last-Match in terms of Rules ?

EDIT 3: I see, you can use "nomatch" when you create the IP Set. I was expecting to see that "invert" Option when building the Rule. Oh well

 

[silverstone]

I don't know if this helps, but I have a rule at the datacenter, host and each VM/CT that drops incoming traffic addresses to ip addresses that are not one of the ones assigned to me. (Define and ipset of your assigned addresses then drop anything not addressed to something in that ipset).

I certainly haven't had to do all that stuff you have, though I do use bridged networking rather than routed.

So I also added your rule (see my EDIT 3 ... I had to create a Rule Set "NOT MY IPs" with "nomatch" enabled. Too bad we cannot do this when creating the Firewall Rule, like I could e.g. with OPNSense ("Invert the sense of the Match"), as then I need to create "MY PUBLIC IPs", "NOT MY PUBLIC IPs", "My Proxmox Public IP", "My OPNSense Public IP", ....

Too early to tell I guess if it's your Rule or a Combination of both (for a few Hours I didn't register anything with tcpdump with the Rules I originally implemented before your Reply).

Now it's been approx. 8 hours and

tmux
tcpdump -Q out -n -w /root/tcpdump.pcap -e not ether host ab:cd:de:ab:cd:de and not ether host 01:23:45:67:89:01 &
tail -f /root/tcpdump.pcap

Did NOT register anything.

 

[silverstone]

Thanks Undergrid, you are right, it was my fault. I have vergotten to switch on in the Hetzner firewall the subnet. Now it's running!!

sorry
Bernd

Not sure if it helps, but you might want to check my reply and @Undergrid Firewall Rule.

In my case the First Abuse was originally automatically solved, just to come back a few days later + manual Reply saying that they would block my server within a few hours if the Problem was not solved.

I'm still not sure whether this is purely a ICMP-Related Flooding (which can, in some circumstances, lead to a Denial of Service, which is probably why they take this VERY seriously, basically all Hosts starting to reply/broadcasts even when not specifically Adressed, thus amplifying the "Attack"), or if it's also TCP/UDP Related. For now the "Drastic" measure of disabling ICMP altogether on Proxmox VE itself seemed to help.

I also added this in order to Harden the System against Attacks (not really Network related though) in /etc/sysctl.d/99-hardening.conf:

# eBPF exposes quite large attack surface and must therefore be restricted
# These sysctls restrict eBPF to the CAP_BPF capability (CAP_SYS_ADMIN on kernel versions prior to 5.8) and enable JIT hardening techniques, such as constant blinding.
kernel.unprivileged_bpf_disabled=1
net.core.bpf_jit_harden=2

# Restricts loading TTY line disciplines to the CAP_SYS_MODULE capability to prevent unprivileged attackers from loading vulnerable line disciplines with the TIOCSETD ioctl,
dev.tty.ldisc_autoload=0

#  The userfaultfd() syscall is often abused to exploit use-after-free flaws. Due to this, this sysctl is used to restrict this syscall to the CAP_SYS_PTRACE capability.
vm.unprivileged_userfaultfd=0

# This only permits symlinks to be followed when outside of a world-writable sticky directory,
# when the owner of the symlink and follower match or when the directory owner matches the symlink's owner.
# This also prevents hardlinks from being created by users that do not have read/write access to the source file. Both of these prevent many common TOCTOU races.
fs.protected_symlinks=1
fs.protected_hardlinks=1

# These prevent creating files in potentially attacker-controlled environments, such as world-writable directories, to make data spoofing attacks more difficult.
fs.protected_fifos=2
fs.protected_regular=2

# Process that run with elevated privileges may still dump their memory even after these settings.
# To prevent them from doing so, set the following via sysctl:
fs.suid_dumpable=0

# Similar to core dumps, swapping or paging copies parts of memory to disk, which can contain sensitive information.
# The kernel should be configured to only swap if absolutely necessary
vm.swappiness=1

# dmesg is the kernel log. It exposes a large amount of useful kernel debugging information, but this can often leak sensitive information, such as kernel pointers.
# Changing the above sysctl restricts the kernel log to the CAP_SYSLOG capability.
kernel.dmesg_restrict=1

# Despite the value of dmesg_restrict, the kernel log will still be displayed in the console during boot.
# Malware that is able to record the screen during boot may be able to abuse this to gain higher privileges. This option prevents those information leaks.
# This must be used in combination with certain boot parameters described below to be fully effective.
kernel.printk=3 3 3 3

# A kernel pointer points to a specific location in kernel memory.
# These can be very useful in exploiting the kernel, but kernel pointers are not hidden by default — it is easy to uncover them by, for example, reading the contents of /proc/kallsyms.
# This setting aims to mitigate kernel pointer leaks. Alternatively, you can set kernel.kptr_restrict=1 to only hide kernel pointers from processes without the CAP_SYSLOG capability.
kernel.kptr_restrict=2


# Performance events add considerable kernel attack surface and have caused abundant vulnerabilities
# This sysctl restricts all usage of performance events to the CAP_PERFMON capability (CAP_SYS_ADMIN on kernel versions prior to 5.8).
# Be aware that this sysctl also requires a kernel patch that is only available on certain distributions.
# Otherwise, this setting is equivalent to kernel.perf_event_paranoid=2, which only restricts a subset of this functionality.
# kernel.perf_event_paranoid=3


# The SysRq key exposes a lot of potentially dangerous debugging functionality to unprivileged users.
# Contrary to common assumptions, SysRq is not only an issue for physical attacks, as it can also be triggered remotely.
# The value of this sysctl makes it so that a user can only use the secure attention key, which will be necessary for accessing root securely. Alternatively, you can simply set the value to 0 to disable SysRq completely.
#kernel.sysrq=4

# ptrace is a system call that allows a program to alter and inspect another running process, which allows attackers to trivially modify the memory of other running programs.
# This restricts usage of ptrace to only processes with the CAP_SYS_PTRACE capability.
# Alternatively, set the sysctl to 3 to disable ptrace entirely.
kernel.yama.ptrace_scope=2

/etc/modprobe.d/hardening-filesystems.conf:

install cramfs /bin/false
install freevxfs /bin/false
install jffs2 /bin/false
install hfs /bin/false
install hfsplus /bin/false
install squashfs /bin/false
install udf /bin/false
install cifs /bin/true
install nfs /bin/true
install nfsv3 /bin/true
install nfsv4 /bin/true
install ksmbd /bin/true
install gfs2 /bin/true

/etc/modprobe.d/hardening-network-protocols.conf:

install dccp /bin/false
install sctp /bin/false
install rds /bin/false
install tipc /bin/false
install n-hdlc /bin/false
install ax25 /bin/false
install netrom /bin/false
install x25 /bin/false
install rose /bin/false
install decnet /bin/false
install econet /bin/false
install af_802154 /bin/false
install ipx /bin/false
install appletalk /bin/false
install psnap /bin/false
install p8023 /bin/false
install p8022 /bin/false
install can /bin/false
install atm /bin/false

/etc/modprobe.d/hardening-other.conf:

install vivid /bin/false
install bluetooth /bin/false
install btusb /bin/false
install uvcvideo /bin/false
install usb-storage /bin/false

/etc/default/grub.d/hardening.cfg:

GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} slab_nomerge init_on_alloc=1 init_on_free=1 page_alloc.shuffle=1 pti=on randomize_kstack_offset=on vsyscall=none debugfs=off lockdown=confidentiality mce=0 quiet loglevel=0"
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} slab_nomerge init_on_alloc=1 init_on_free=1 page_alloc.shuffle=1 pti=on randomize_kstack_offset=on vsyscall=none debugfs=off lockdown=confidentiality mce=0 quiet loglevel=0"

I had found this online which looks quite Interesting as a Guide for hardening Linux Systems:
https://dounpct.medium.com/patch-and-hardening-rke-cluster-ff8811acb5af

Specifically this Tool that makes an Audit of the System and proposes Reccomendations.

Anybody has any Experience with it ?

https://packages.cisofy.com/ - Lynis client (Community Edition) - https://github.com/CISOfy/lynis

 

[wowo]

I don't know if this helps, but I have a rule at the datacenter, host and each VM/CT that drops incoming traffic addresses to ip addresses that are not one of the ones assigned to me. (Define and ipset of your assigned addresses then drop anything not addressed to something in that ipset).

I certainly haven't had to do all that stuff you have, though I do use bridged networking rather than routed.

I think this might not solve the problem as the traffic reaching your machine - even if being dropped - was intended for another machine, so the other customer would have package loss.

 

[silverstone]

I think this might not solve the problem as the traffic reaching your machine - even if being dropped - was intended for another machine, so the other customer would have package loss.

Well, with all due respect, that is "his/her Problem" and/or (most likely) some weird Stuff going on on the part of Hetzner.

Personally I don't want that they take my Server Down because their Server (plus possibly my Server) was badly configured. Would you accept that ?

And, at the same time, we can only control what it's in our Responsibility. I have no control on who manages their Server, configure it, what they do with it, etc.

Would I love Hetzner to include all of the Reccomendation that I found and am posting on this Thread, on their Wiki ? ABSOLUTELY. I believe knowledge brings power, and everybody should configure their Server correctly (with at least a good 90% Hardened Configuration or so). I'm not a Sysadmin by Training and I am always learning new stuff.

But Hetzner, while having many Wiki / Community Tutorials, at the end of the Day says "we cannot help you configure each/your operating System". Fair to some extent ... if everybody would contact them for that, they would lose a lot of Money in terms of customer Support. Like ... if they provide the "Solution", then they must also take Responsibility for it and Support, if stuff goes wrong.

Nevertheless I believe it would be a good idea to include these reccomendation in a Wiki Page or Community Tutorial or something, like "we take no responsibility, but here is a good hardening baseline config for your servers, use at your own risk or something like that". Would avoid lots of Frustration with people getting MAC Abuse Notifications for no apparent reason/fault of their own (other than their Server responding to traffic that somebody else is Generating).

I think I'll just do a GitHub Repository for this and start collecting things there. At the end of the Day, I think all of these Things can also be deployed in my Homelab Environment (with some Exceptions with regards to IPv4/IPv6 Routing, for some Systems).

From my Perspective, I believe we all have our duties/responsibilities with regards to:
- Harden our System so that we don't become a Target ourselves (Inbound Responsibility)
- Do NOT participate in DDOS / ICMP Flood / etc when other Servers try to "trigger" us (Outbound Responsibility)

So we can harden our system Inbound (mainly via Firewall, either Hetzner's Firewall, Proxmox VE Firewall, or a combination of both [what I use]).

To harden the System Outbound I believe that the /etc/sysctl.conf Parameters that I suggested go a VERY LONG WAY in solving the Issue.

I re-enabled Ping (net.ipv4.icmp_echo_ignore_all=0 and net.ipv6.icmp.echo_ignore_all=0) so that Server Monitoring works again, and hardened the System some more.

Updated /etc/sysctl.d/99-hetzner.conf

# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# WE MUST ALSO SPECIFY EXPLICITELY eth0 HERE (maybe also vmbr0)
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

# Disable IPv4 multicast igmp report on local link plug
net.ipv4.igmp_link_local_mcast_reports=0

# Ignore IPv4 ICMP echo packets sent to a broadcast address
net.ipv4.icmp_echo_ignore_broadcasts=1

# Ignore IPv6 ICMP echo packets sent to anycast & multicast address
# This works very similarly to net.ipv4.icmp_echo_ignore_broadcasts
net.ipv6.icmp.echo_ignore_anycast=1
net.ipv6.icmp.echo_ignore_multicast=1

# This setting avoids filling up log files with unnecessary error messages coming from invalid responses to broadcast frames.
net.ipv4.icmp_ignore_bogus_error_responses=1

# Don't ignore directed pings
# This will also make PING the Proxmox VE Host Impossible !
net.ipv4.icmp_echo_ignore_all=0
net.ipv6.icmp.echo_ignore_all=0

# Disables the acceptance of ICMP redirect messages.
# Make sure no one can modify the routing Table
# Ignore IPv4 ICMP redirect messages
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
net.ipv4.conf.vmbr0.accept_redirects=0

# Ignore IPv6 ICMP redirect messages
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.eth0.accept_redirects=0
net.ipv6.conf.vmbr0.accept_redirects=0

# Accepting 'secure' ICMP redirects (from those gateways listed as default gateways) has few legitimate uses.
# It should be disabled unless it is absolutely required.
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.eth0.secure_redirects=0
net.ipv4.conf.vmbr0.secure_redirects=0

# A node should not send out IPv4 ICMP redirects, unless it acts as a router.
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.vmbr0.send_redirects=0

# Do NOT act as a Router
# IP forwarding should only be enabled on systems acting as routers.

# Disable IPv4 Packet Forwarding
net.ipv4.ip_forward=0

net.ipv4.conf.default.forwarding=0
net.ipv4.conf.all.forwarding=0
net.ipv4.conf.eth0.forwarding=0
net.ipv4.conf.vmbr0.forwarding=0

# Disable IPv6 Packet Forwarding
net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.forwarding=0
net.ipv6.conf.eth0.forwarding=0
net.ipv6.conf.vmbr0.forwarding=0

# No source routed packets here
# This setting disables the acceptance of packets with the SSR option set in the IPv4 packet header.
# Packets that use Source Routing will be rejected.
# This prevents IP packet redirection, that is redirection to a host behind a firewall, that is not directly reachable otherwise.
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.eth0.accept_source_route=0
net.ipv4.conf.vmbr0.accept_source_route=0

# Do not accept packets with SRR option
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.eth0.accept_source_route=0
net.ipv6.conf.vmbr0.accept_source_route=0

# Disable IPv6 Router Advertisements, since they can result in a man-in-the-middle attack
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.eth0.accept_ra=0
net.ipv6.conf.vmbr0.accept_ra=0

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.default.log_martians=1
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.eth0.log_martians=1
net.ipv4.conf.vmbr0.log_martians=1

# Turn on syncookies for SYN flood attack protection
# This enables TCP SYN Cookie Protection for IPv4 and IPv6.
net.ipv4.tcp_syncookies=1

# Turn on reverse path filter in Strict Mode
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.eth0.rp_filter=1
net.ipv4.conf.vmbr0.rp_filter=1

# Disable forwarding of all multicast packets on all interfaces
net.ipv4.conf.default.mc_forwarding=0
net.ipv4.conf.all.mc_forwarding=0
net.ipv4.conf.eth0.mc_forwarding=0
net.ipv4.conf.vmbr0.mc_forwarding=0

net.ipv6.conf.default.mc_forwarding=0
net.ipv6.conf.all.mc_forwarding=0
net.ipv6.conf.eth0.mc_forwarding=0
net.ipv6.conf.vmbr0.mc_forwarding=0

# Drop RST packets for sockets in the time-wait state
net.ipv4.tcp_rfc1337=1

# Disable TCP SACK
net.ipv4.tcp_sack=0
net.ipv4.tcp_dsack=0
net.ipv4.tcp_fack=0

# TCP timestamps also leak the system time and should therefore be disabled
net.ipv4.tcp_timestamps=0

# Enable IPv6 Privacy Extensions
net.ipv6.conf.default.use_tempaddr=2
net.ipv6.conf.all.use_tempaddr=2
net.ipv6.conf.eth0.use_tempaddr=2
net.ipv6.conf.vmbr0.use_tempaddr=2

# The TCP SYN backlog defines the number of SYN packets that are queued for further processing.
# Once the queue limit is exceeded, all new incoming SYN-packets are dropped and new TCP connections will not be possible (or the SYN cookie protection kicks in)
# Increasing this value improves the protection against TCP SYN flood attacks.
net.ipv4.tcp_max_syn_backlog=4096

#########################################################
############ NEWLY INTRODUCED ON 2024-05-17 #############
#########################################################

# Number of global unicast IPv6 addresses can be assigned to each interface
net.ipv6.conf.default.max_addresses=1
net.ipv6.conf.all.max_addresses=1
net.ipv6.conf.eth0.max_addresses=1
net.ipv6.conf.vmbr0.max_addresses=1

# Number of neighbor solicitations to send out per address
net.ipv6.conf.default.dad_transmits=0
net.ipv6.conf.all.dad_transmits=0
net.ipv6.conf.eth0.dad_transmits=0
net.ipv6.conf.vmbr0.dad_transmits=0

# Router advertisements can cause the system to assign a global unicast address to an interface
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.eth0.autoconf=0
net.ipv6.conf.vmbr0.autoconf=0

# Setting controls whether the system will accept Hop Limit settings from a router advertisement
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.eth0.accept_ra_defrtr=0
net.ipv6.conf.vmbr0.accept_ra_defrtr=0

# Learn prefix information in router advertisement
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.eth0.accept_ra_pinfo=0
net.ipv6.conf.vmbr0.accept_ra_pinfo=0

# Do not accept Router Preference from RA
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.eth0.accept_ra_rtr_pref=0
net.ipv6.conf.vmbr0.accept_ra_rtr_pref=0

# Number of Router Solicitations to send until assuming no routers are present
net.ipv6.conf.default.router_solicitations=0
net.ipv6.conf.all.router_solicitations=0
net.ipv6.conf.eth0.router_solicitations=0
net.ipv6.conf.vmbr0.router_solicitations=0

Slightly updated /etc/sysctl.d/99-hardening.conf

# eBPF exposes quite large attack surface and must therefore be restricted
# These sysctls restrict eBPF to the CAP_BPF capability (CAP_SYS_ADMIN on kernel versions prior to 5.8) and enable JIT hardening techniques, such as constant blinding.
kernel.unprivileged_bpf_disabled=1
net.core.bpf_jit_harden=2

# Restricts loading TTY line disciplines to the CAP_SYS_MODULE capability to prevent unprivileged attackers from loading vulnerable line disciplines with the TIOCSETD ioctl,
dev.tty.ldisc_autoload=0

#  The userfaultfd() syscall is often abused to exploit use-after-free flaws. Due to this, this sysctl is used to restrict this syscall to the CAP_SYS_PTRACE capability.
vm.unprivileged_userfaultfd=0

# This only permits symlinks to be followed when outside of a world-writable sticky directory,
# when the owner of the symlink and follower match or when the directory owner matches the symlink's owner.
# This also prevents hardlinks from being created by users that do not have read/write access to the source file. Both of these prevent many common TOCTOU races.
fs.protected_symlinks=1
fs.protected_hardlinks=1

# These prevent creating files in potentially attacker-controlled environments, such as world-writable directories, to make data spoofing attacks more difficult.
fs.protected_fifos=2
fs.protected_regular=2

# Process that run with elevated privileges may still dump their memory even after these settings.
# To prevent them from doing so, set the following via sysctl:
fs.suid_dumpable=0

# Similar to core dumps, swapping or paging copies parts of memory to disk, which can contain sensitive information.
# The kernel should be configured to only swap if absolutely necessary
vm.swappiness=1

# dmesg is the kernel log. It exposes a large amount of useful kernel debugging information, but this can often leak sensitive information, such as kernel pointers.
# Changing the above sysctl restricts the kernel log to the CAP_SYSLOG capability.
kernel.dmesg_restrict=1

# Despite the value of dmesg_restrict, the kernel log will still be displayed in the console during boot.
# Malware that is able to record the screen during boot may be able to abuse this to gain higher privileges. This option prevents those information leaks.
# This must be used in combination with certain boot parameters described below to be fully effective.
kernel.printk=3 3 3 3

# A kernel pointer points to a specific location in kernel memory.
# These can be very useful in exploiting the kernel, but kernel pointers are not hidden by default — it is easy to uncover them by, for example, reading the contents of /proc/kallsyms.
# This setting aims to mitigate kernel pointer leaks. Alternatively, you can set kernel.kptr_restrict=1 to only hide kernel pointers from processes without the CAP_SYSLOG capability.
kernel.kptr_restrict=2


# Performance events add considerable kernel attack surface and have caused abundant vulnerabilities
# This sysctl restricts all usage of performance events to the CAP_PERFMON capability (CAP_SYS_ADMIN on kernel versions prior to 5.8).
# Be aware that this sysctl also requires a kernel patch that is only available on certain distributions.
# Otherwise, this setting is equivalent to kernel.perf_event_paranoid=2, which only restricts a subset of this functionality.
# kernel.perf_event_paranoid=3


# The SysRq key exposes a lot of potentially dangerous debugging functionality to unprivileged users.
# Contrary to common assumptions, SysRq is not only an issue for physical attacks, as it can also be triggered remotely.
# The value of this sysctl makes it so that a user can only use the secure attention key, which will be necessary for accessing root securely. Alternatively, you can simply set the value to 0 to disable SysRq completely.
#kernel.sysrq=4

# ptrace is a system call that allows a program to alter and inspect another running process, which allows attackers to trivially modify the memory of other running programs.
# This restricts usage of ptrace to only processes with the CAP_SYS_PTRACE capability.
# Alternatively, set the sysctl to 3 to disable ptrace entirely.
kernel.yama.ptrace_scope=2

# Forbid creating Kernel Core Dumps
kernel.core_pattern=|/bin/false

Plus, as I mentioned, implementing @Undergrid Firewall Rule Reccomendation.

So far, knock on Wood, it's been 3.5 days with tcpdump running and nothing logged so far (with a different MAC Address than those that I am allowed by Hetzner) .

@proxmox staff: would you be interested in you/us creating such a Wiki Page on your Wiki ?

 

[wowo]

Personally I don't want that they take my Server Down because their Server (plus possibly my Server) was badly configured. Would you accept that ?

No of course not. Can you show your current /etc/network/interfaces again?

 

[silverstone]

No of course not. Can you show your current /etc/network/interfaces again?

Sure.

Relevant Part (excluded are the DMZ Zones which will be managed by OPNSense) and of course redacted IP for Privacy.

auto lo
iface lo inet loopback

# Disabled the "auto" Part
#auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
    hwaddress XXXXXXXXXXXXXX
    address xx.yyy.yyy.zzz
    netmask 255.255.255.255
    gateway xx.yyy.yyy.zzz
    pointopoint xx.yyy.yyy.zzz
    bridge-ports eth0
    bridge-stp off
    bridge_waitport 0
    bridge-fd 0
    bridge-disable-mac-learning 1
    bridge-unicast-flood off
    bridge-multicast-flood off
    bridge-vlan-aware yes
    bridge-vids 2-4096
    pre-up ip addr flush dev eth0
    post-up ip addr flush dev eth0

And some Comments I had Kept in the File as well (moved further up ... I think I will NEVER allow Proxmox-VE to change these Configuration from the Web GUI):

# Comments moved at the top
# Proxmox VE doesn't like inline Comments
# auto vmbr0
# IPv4
#    Main IP address of the server
#    Netmask 255.255.255.255 (/32) independent from the real subnet size (e.g. /27)
#    What is described as Gateway on Hetzner Robot
#    Bridge Options
# IPv6
  # One Address within the Server-Assigned IPv6 Subnet
  # The "default" address was (after Installation) 1234::5678:9012:3456:7890
  # Netmask  (this works correctly)
  # Gateway 

# https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_disabling_mac_learning_on_a_bridge
# Once enabled, Proxmox VE will manually add the configured MAC address from VMs and Containers to the bridges forwarding database to ensure that
# guest can still use the network - but only when they are using their actual MAC address
#
#        bridge-disable-mac-learning 1
#
# Furthermore bridge-vlan-aware need to be enabled (even if you don't use any vlan)
# eth0 should disable prosmisc mode with this config.

 

[wowo]

I am wondering why you set the netmask to /32.
Is your secondary IP in a subnet different from your primary IP?

 

[silverstone]

I am wondering why you set the netmask to /32.
Is your secondary IP in a subnet different from your primary IP?

No that's just the primary IP Configuration.

Setup was done according to Hetzner own Documentation: https://docs.hetzner.com/robot/dedicated-server/network/net-config-debian-ubuntu/#ipv4

 

[silverstone]

But I just discovered something weird with my OPNSense VM.

For some reason:

1. I cannot update OPNSense anymore
2. From OPNSense I cannot Ping "OUT" to e.g. Google DNS Servers 8.8.8.8 (I have a Uptime-Kuma instance at Home, and Ping "IN" works correctly)
3. Node -> OPNSense VM -> Firewall -> Log only shows "no content" all of a sudden (it used to work previously !)

EDIT 1: the "nomatch" option in PVE Firewall doesn't do what I thought it did. And I'm NOT the only one apparently ... https://forum.proxmox.com/threads/inverted-ip-range-matching.109704/#post-472188. Indeed I was getting a ton of logs on the Node, and nothing in the VM ...

 

[silverstone]

I'm getting crazy with this. Lost yet another day and it's nowhere closer to working reliably.

I can openvpn into the OPNSense VM just fine, but other than that, nothing seems to work. Ping from OPNSense to Outside world is completely broken, so is traceroute etc. I also disabled Proxmox Firewall on that VM/Interface, but it made no Difference .

 

[wowo]

Debug it step by step...

Run a ping from your VM to the outside world. Now check where it gets lost. On the host, sniff traffic on the bridge that your VM is connected to, to see if the correct MAC for that IP shows up or not. If it reaches the bridge, you can be sure, that the problem is on host or network-level.

tcpdump -i vmbr0 ether host xx:xx:xx:xx:xx:xx